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primarily upon two things: they either save time or they increase the 
literature coverage. This study attempts to measure these factors 
from data gathered by a questionnaire which was sent to a selected 
group of industrial chemists. Thus the habits of a statistically 
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semiguantitative determination of the cost-effectiveness of 
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COST EFFECTIVENESS OF INFORMATION SYSTEMS 



In May, 1968, a subcommittee of the Committee on 
Corporation Associates was established "to study the economics 
of chemical information activities in industry". This was 
done in response to a direct request from the ACS Committee 
on Chemical Abstracts Service for such a study and to meet 
a general ..need by the ACS for a way to measure the cost- 
effectiveness of information services. A preliminary report 
of the findings of the subcommittee was presented to the 
Corporation Associates in November, 1968. This is the final 
report of the subcommittee. 

Several previous groups have instituted studies of 
the economics of information systems. Two major current activi- 
ties are being carried on, by the Organization for Economic 
Cooperation and Development and by the Division of Chemistry 
and Chemical Technology, National Research Council. These 
studies, as well as some of the earlier ones, have focused 
on the more manageable problem of developing methods for the 
measurement of costs of information systems and services. As 
far as the subcommittee knows, there have as yet been no reports 
by any of these groups on cost-effectiveness, and the earlier 
studies were apparently abandoned without reaching any con- 
clusions in the area of cost-effectiveness. 

Two reports of interest to the present study have been 
published. Martyn^^ carried on an extensive study of literature 
searching by research scientists, to obtain data on the habits of 

research people in carrying on searches before Instituting new 

» 

projects and the amount of duplication and wastage of time and 

effort in research projects, as indicated by the later discovery 

( 2 ) 

of important references. Brockis and Cole have published a 
more limited survey of the incidence of late discoveries of 
important references. 
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"Literature Searching by Research 
arch Department, London, 1964. 
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In view of the apparent adequacy of current efforts 
and published reports in the cost-measurement area, and the 
almost complete lack of even qualitative guides for determin- 
ing the effectiveness of information systems, the subcommittee 
decided to confine its activities to the latter aspect of tne 
economics problem. The subcommittee was fully aware of the 
great difficulty it would have in attempting to establish any 
reasonably satisfactory quantitative measure of effectiveness, 
but felt that even a very rough estimate would be better than 
nothing — which is the present state of affairs. 

Clearly there are a number of ways in which an 
information system might be "ef feet ive" • It can save time, 
by improving access to the literature. It can make available 
a greater body of literature, within a given time. This, in 
turn, leads to the possibility that an important new idea may 
be stimulated. It can help prevent duplication of research 
efforts. It can provide specific information required for a 
research project* or background information needed for entry 
into a new field. These and other aspects of "e f f ect iveness " 
were considered, and it was concluded that some may be suscep- 
tible to at least a semiquant it ative study. 

Basically, the values of a service depend primarily 
upon two things — they either save time or they increase the 
literature coverage. Obviously these are not entirely indepen- 
dent. A person might, at one extreme, set in effect a time 
limit for himself and cover as much literature as he could in 
that time, and, at the other extreme, devote whatever time it 
took to cover a given amount of literature; or, he could elect 
an intermediate course, saving some time and extending his 
literature coverage somewhat. These factors — time saving 
and greater literature coverage — may be measurable, and 
this is what the present study attempted to do. More nebulous 
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values, such as the worth of a brilliant idea that results 
from improved access to the literature, appear to be impos- 
sible to measure and this wa3 not attempted. This may suggest, 
however, that the values determined in the manner used for this 
study could constitute a lower bound. 

The avoidance of duplication of effort, which was the 
principal factor studied by Martyn and by Brockis and Cole, is 
also fairly easily relatable to value, and is in part an alter- 
native approach. If, as hypothesized above, the user keeps 
constant his total time devoted to the literature and succeeds 
in increasing his literature coverage by the use of a particular 
service, he may benefit by being led to a new idea, or by avoid- 
ance of duplication and wastage of research effort through 
discovery of relevant literature. If, on the other hand, the 
user covers all the relevant literature, no matter what the 
time required may be, no benefits of these, sorts will result 
from the availability of the service, and the value will be 
strictly in terms of time saved and the worth of that time. 

As noted above, an intermediate result will also be expected, 
frequently . 

It was noted also that the use of technical literature 
falls into two broad categories, described briefly as "current 
awareness" and "searching". The first of these relates to the 
need for a scientist or technologist to keep up with the current 
literature, mainly in his own specialty and in closely related 
fields, but also to some extent in remoter fields, in search of 
new ideas and to gain perspective. The second general category 
is the mor°. or less thorough coverage of the literature, in 
search of specific information or to provide a survey of all 
the work in a given area. 

In order to obtain data of the kind described, a 
questionnaire was prepared and sent to a selected group of 
industrial chemists. The first version was sent to people 
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in the companies represented on the Committee on Corporation 
Associates, and an amended and revised version was later sent 
to a much larger group of people, in all the companies which 
are Corporation Associates. In each mailing, the company 
representative on the Corporation Associates' was asked to 
solicit responses from ten of his associates, selected in as 
random a manner as was convenient from among his company’s 
professional "research 11 chemists. This would be expected 
to include many development chemists, but was intended to 
exclude a number of people such as technical sales, informa- 
tion specialists, etc. The preliminary survey went to 150 
chemists, with 135 usable responses received. The second 
went to 2790 chemists, with 1394 usable responses. The sample 
sizes are generally large enough to permit statistically 
meaningful conclusions to be drawn. 



The second questionnaire is attached. (The first 
version wr.s similar in concept and differed only in its 
details.) The first page asked for background information 
on the respondent arid his company, to permit a later breakdown 
of the results according to the sub catego r ies indicated. The 
second page asked for data on the use of information services. 
Sources of information were divided into seven groupings: 
Primary sources, Books and reviews. Abstract services, Title 
services, Science Citation Index, Patent services, and Com- 
puterized information services. The respondent was asked to: 

(1) indicate which groups of services were available to him; 

(2) estimate the number of hours per week spent using each of 
the groups of services, for both Current Awareness and for 
Searching (separately); (3) estimate the number of hours per 
week he saved through the use of each of the categories of 
services available to him, again for both the Current Awareness 
and Searching functions (omitting Primary Sources, and Books 
and Reviews) ; (4) estimate the percentage increase in his 
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coverage of useful literature through the use of each of the 
categories of services available to hira> for Current Awareness 
and for Searching. 

The profile data for the respondents and the^r 
companies are as follows: 



Age 




Highest Degree 




Under 30 


214 


None 


21 


31-40 


591 


Bachelor 1 s 


36 7 


41-50 


389 


Mas ter T s 


230 


Over 50 


196 


PhD/DSc 


768 


(no reply 


4) 


Other 
(no reply 


7 

1) 



Years of 


Experience 


Number of People Reporting 
to the Respondent 


Under 5 


268 


None 


268 


5-10 


309 


1-3 


612 


10-20 


515 


4-10 


322 


20-30 


215 


11-20 


125 


Over 30 


84 


Over 20 


66 


(no reply 


3) 


(no reply 


1) 



* Field of Specialization Field of Activity 



Organic 


650 


Research 


962 


Physical 


261 


Development 


333 


Analytical 


232 


Administration 


72 


Inorganic 


146 


Other 


71 


Biological 


120 






Medicinal 


122 






Plas tics 


181 






Petroleum 


87 






Engineering 


139 






Other 


2 34 






(^Multiple 


responses allowed) 
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^Principal Business 


Size of Company 




of Comp 


any 


- annual sales 




Chemicals 


599 


Under $10 million 


60 


Plastics 


160 


$10-100 million 


30 8 


Rubber 


65 


Over $ iGG uizlliuii 


990 


Drugs 


211 


(no reply 


14) 


Electrical 


76 






Metals 


85 






Petroleum 


121 






Other 


544 






(no reply 


2) 






(^Multiple r 


espouses 


allowed) 





The availability of categories of information services was 
found t o be : 



Abstract services 92% 
Title services 76% 
Science Citation Index 20% 
Patent services 84% 



Computerized information services 32% 



It was apparent that a number of respondents did not really 
know whether or not they had Science Citation Index available. 
It was also evident that the computerized information services 
were more likely to be available in large companies than in , 
small ones (38% in companies with sales over $100 million vs. 
17% in those under $100 million) , 

The percentages of people using each of the services 
available to them were as follows: 



Current Either 





Awareness 


Searching 


Usage 


Abstract services 


67% 


80% 


9 3% 


Title services 


52% 


29% 


62% 


Science Citation Index 


11% 


21% 


24% 


Patent services 


6 3% 


45% 


68% 


Computerized information 


42% 


39% 


60% 



services 




8 



7 



Tha low usage of Science Citation Index undoubtedly reflects a 
general unfamiliarity with this service, since users of it find 
it to be highly effective. This suggests that many people not., 
using Science Citation Index woulu^find it profitable to learn 
to use it. It is perhaps somewhat surprising that the percentage 
using available computerized services is not higher. 

The time spent by chemists in their use of information 
sources is of considerable interest. The data indicate that this 
time is rather higher than is generally thought: 





Current 






Awareness 


Searching 




(hrs/week) 


(hrs/week) 


Primary sources 


4.2 


1.6 


Books and reviews 


1.5 


1.2 


Abstract services 


0.8 


1.1 


Title services 


0.4 


0.2 


Science Citation Index / 


0.1 


0.1 


Patent services 


0.6 


0.4 


Computerized services 


0.3 


0.2 



The numbers for Primary sources and for Books and reviews are 
averages for all respondents, as it was assumed that everyone 
had access to these. The figures for the other categories are 
averages for all people who have access to these services, 
including those who do not use them. If the averages are 
taken only for those using the services, the results are: 



Current 

Awareness Searching 

(hrs /week) (hrs/week) 



Abstract services 1.3 
Title services 0.9 
Science Citation Index 0.6 
Patent services 0.9 
Computerized services 0.8 



1.4 

0.7 

0.5 

0.8 

0.7 



The averages cannot be added, in either of the above tables, 
because the separate figures refer to somewhat different 
opulat ions . 
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Averages computed for all 1394 respondents, whether 
or not they have a particular service, give a figure for the 
average time spent using information sources by all industrial 
chemists. These numbers are 7.5 hrs/week for Current Awareness 
and 4.3 hrs/week for Searching, for a total time of 11.8 hrs/ 
week. No important differences between people in large and 
small companies appeared. There were some differences between 
people in various subfields of chemistry. Organic chemists and 
biological/medicinal chemists spend more time with information 
sources than do chemists as a whole; the figures for organic 
chemists total 13.2 hrs/week, and for biological /medicinal 
chemists 13.7 hrs/week. Research people spend more time on 
information than do non-research chemists, totaling 12.6 hrs/ 
week as compared with 10.2 hrs/week for non-research chemists. 
People in chemical companies spend slightly more time than the 
average chemist surveyed, 12.3 hrs/week. People in drug com- 
panies spend considerably more time, 13.3 hrs/week. 



Estimates of time saved, through the use of a 
particular information service, are of course difficult for 
anyone to make. A few respondents pointed out (correctly 
of course) that these quantities were "impossible to estimate" ; 
most respondents made the attempt, however. The hope was that 
there would be no systematic tendency to overestimate or under- 
estimate this quantity, and that the averages would therefore 
have at least a rough quantitative validity. In full realization 
of the very limited accuracy of the data, then, the following 
results are presented: 

Current 

Awareness Searching 

(hrs/week) (hrs/week) 



Abstract services 3.2 ' 3.9 
Title services 2.6 1.8 
Science Citation Index 1.6 1.3 
Patent services 2.0 2.1 
Computerized services 2.5 3.1 




averages 



are for people actually using the service. 
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A measure of the effectiveness of each service can 
be obtained by comparing these estimates of time saved with 
the earlier figures reported for time spent. Thus the ratio 
of the two gives an estimate of the number of hours saved for 
each hour spent, with a given service. For example, Abstract 
services for Searching are said to save 4.0 hrs/week by the 
expenditure of 1.4 hrs/week; this is 2.9 hours per hour spent. 

(Obviously the time saved, or the effectiveness, of a particular 

\ 

service will also depend to some extent upon what other services 
are used, etc.) By this criterion the order of effectiveness of 
information services is: 



Current Awareness (hrs/week) Searching (hrs/week) 



Computerized services 


3.1 


Computerized services 


4.8 


Title services 


2.9 


Abstract services 


2.8 


Science Citation Index 


2 . 7 


Science Citation Index 


2.7 


Abs tract services 


2.5 


Title services 


2.6 


Patent services 


2.1 


Patent services 


2.6 



Small differences in these numbers are not meaningful in measuring 
order of effectiveness. 

The questionnaire also asked for an estimate of the 
percentage increase in the coverage of useful literature by the 
use of information service. The results for Current Awareness 
were: Abstract services 90%, Title services 90%, Computerized 

services 90%, Patent services 70%, and Science Citation T ndex 
20%. For Searching, the results were: Abstract services 140%, 

Computerized services 100%, Title services 70%, Patent services 
70%, and Science Citation Index 50%. It is felt that these data 
may be rather unreliable, as a number of respondents indicated 
considerable difficulty in making some of the estimates. For 
example, the increased coverage is probably more apparent to 
the user of Abstract services for Searching than is Science 
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(3) 

Citation Index, although other studies of this comparison 
indicate that for short searches Science Citation Index is 
actually more efficient, and they are about equal in the long 
run . 

Finally, as an exercise to see what value might be 
placed upon a particular category of service, we can use the 
data for Computerized services. A total of 440 respondents 
have them available (less than half of these people actually 
use the services, as noted above). They say that their use 
of Computerized services results in a time saving of 996 hours 
(455 for Current Awareness, 541 for Searching). These numbers 
thus indicate that the provision of Computerized services to 
an average group of industrial chemists will result in a time 
saving of 2.3 hrs/week (996/440), averaged over all the staff, 
i.e., including those who do not use the services as well as 
those who do u Obviously the figure would be higher if more of 
the staff used these services. Thus the "value” of Computerized 
services for, say, a staff of 100 chemists costing $20/hour 
(including overhead) for a year would be 100 x 2.3 x 20 x 52 = 
$250 ,000/year . It can be argued that this is a lower bound to 
the value, since no allowance has been made for the value of 
increased literature coverage. On this basis, it might be 
legitimate to use this "value" in a comparison with the costs 
for computerized services, to determine cost-effectiveness. 

Similar calculations could be made for the other 
catagories of services. The time saved, averaged over all 
staff members, is as follows: Abstract services 5.4 hrs/week, 

Title services 1.9 hrs/week, Science Citation Index 0.4 hrs/ 
week, and Patent services 2.2 hrs/week. Again, these numbers 
represent a combination of two factors, one being the fraction 
of the people using the service and the other the estimate of 
time saved by those actually using it. Clearly, low fractional 
usage has drastically reduced some of these numbers. 



! 3) C. c. Spencer, American Documentation, April 1967. 
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In summary, we have explored the habits of a 
statistically meaningful sample of industrial chemists in 
their use of information services, and have developed a 
crude model for at least a semiquantitat ive determination 
of the cost-effectiveness of information systems. 



N. B. HANNAY (Chairman) 
May 20, 1969 
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